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• Objectives 

— Reveal foundational lessons, best practices and real-world 
perspectives about how trust and reliance depend on cultural 
and organizational factors and automation capability 

- Synthesize and integrate results to develop a set of questions 
for further research leading to more trustable automation 

• Automatic Ground Collision Avoidance System (Auto-GCAS) 
as the context for case study 

— Contemporary, unique, and of great public interests, especially 
for integration of UAS/UAV into the National Airspace System 

- Projected to save lives and money with 2014 F-16 deployment 

- Development spans 3 decades 

- Research team has access to key individuals and organizations 

• Timeline for project completion: 18 months 
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Best Practice Solution 
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• Utilize an extended case methodology that 
combines grounded theory and traditional 
research 

• Adapt methodology in response to challenges 

• Be culturally competent in regards to 
participants 

• Working with key personnel who facilitated 
access and served as informants 
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Key Stakeholders 
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• End Users: F-16 Pilot Community 

• Team that conceived, designed, built, and 
tested Auto-GCAS 

— NASA Armstrong (formally Dryden) Flight 
Research Center 

— Air Force: Flight Test Center, AF Research 
Lab 

— Lockheed Martin 

• Office Under the Secretary of Defense 

• Air Force Office of Scientific Research: 
funded case study 
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Project Requirements 
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• Have sound understanding of the historical 
development of Auto-GCAS 

• Confidentiality of participants and information 

• Sensitivity to bureaucracies, politics, and 
professional environment 

• Cultural and organizational competency 

• Technical understanding of Auto-GCAS 

• Timeline of completion (18 months) 
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Methodology 


Extended Case Method(ology) 


Grounded Theo[y, Ethnographic 



Stories 
from the 
field 



Surveys 




Collective story (key 
events, persons, meetings 
—generating hypotheses) 


Traditional Research 



Theory Revising 
Comparing Lee & See Model to 
ethnographic data model 


Hypothesis- 
testing (e.g., Lee 
& See Model) 
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Methods used and how 
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• Primary sources (grounded theory/ethnography) 

— Questionnaires and surveys 

— Interviews 
— Field notes 
— Observations 

• Secondary sources (traditional research) 

— Literature review (including internal documents, videos, and 
reports) 

• Data collected was coded using NVivo to extract emergent 
themes and theories 

• Theories generated from both sources were compared to 
see if they converge or diverge 

• Aggregated hypothesis are then used to revise existing 
theories 
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Challenges 
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• Project-related 

— Difficult to capture cultural and organizational factors 

- Limited prior literature in cultural and organizational factors on 
trust in automation 

- Time limitations of project 

— Limited access to personnel and confidential information 

- Busy schedules of participants 

- Participants are remotely located 

- Sensitive nature of politics, bureaucracy, and hierarchy of 
organizations 

• Research team-related 

— Must have good understanding of a highly technical topic 

- Unfamiliar with the cultures of the participants 

— Must sustain team effectiveness when research team members 
have diverse education, time commitments, and training 
backgrounds 
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• Project-related strategy 

— Multi-pronged approach using different methods to capture 
cultural and organizational factors 

• Requested and received training from key informants 

• Gained general background knowledge from literature 

• Immersed research team into the various cultures via observations and 
field notes 

— Used qualitative methods to fill in gaps from literature review 

- Utilized adaptable and agile methodology to account for time 
limitations of the project 

- Key personnel assisted in gaining entree to participants, research 
sites, information and insights (cultural and organizational) 

- Modified 2-hour interview into 1-hour online convenient survey 
and 1-hour follow up interview to address participants' 
schedules 

- Made extensive use of Skype, phone, and availability of 
participants travel schedule in addition to traveling to conduct 
face-to-face interviews 



Strategies to Address Challenges 
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• Team-related strategy 

— Hone technical knowledge by reading technical 
papers and listening to video-recorded 
explanations of the technology 

— Continuously immerse research team into 
participant-rich environments to observe cultural 
and organizational factors in addition to reading 
literature 

- Created a project orientation guide and 
instruction manuals for new research assistants 
and the research team's general knowledge 
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Lessons Learned (1/2) 
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• Extended case study methodology allowed for 
flexibility and was effective 

— Adapt interviews and questionnaire based on 
circumstances of availability of participants and 
opportunities presented 

— Extend field observations to include workplace, 
formal and informal gatherings to immerse into 
the culture of experimental test pilots 

— Add survey to capture current opinions of the 
larger test pilot community 
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Lessons Learned (2/2) 
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• Key personnel assisted in entree and were key 
informants 

— Facilitate team visits to bases, participant 
recruitment & key connections 

— Establish credibility and trustworthiness of team 

— Despite key personnel assistance, research team is 
not able to get participation for all targeted groups 

• Cultural competency 

— Having a good understanding of participant culture 
facilitates communication between the participants 
and the researcher 

— Positive communication creates respect, rapport, 
and trust of participants 
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Conclusions 
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• Team developed a unique and eclectic set of 
qualitative and quantitative methodologies, 
which are adaptable to the study's challenges 

• Extended case study approach was effective in 
collecting data in a military and sensitive 
environment, particularly for researching non- 
technology related factors in trust development 
of automation 

• Methodological framework can be used to study 
other technological systems in similar 
environments 
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Thank You! 
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